Establishing Item Uniqueness for Automatic Item Generation

Objective The growing popularity of Automatic Item Generation (AIG) can be attributed to the increasing demand for the production of large pools of operational test items. AIG is an algorithmic way of generating assessment tasks which combines cognitive theories, psychometric practices and computer technologies. The outcome of this algorithmic transcriptions of assessment task is 1986). The item modeling process requires the identification of features or elements within the assessment task so that these elements can be manipulated to produce large set of items (Gierl et al., 2012). These generated items may or may not be similar to one another and thus the distinctiveness among generated items is largely unknown. Consequently it is important to develop a measure to quantify the similarity among the generated items so that the distinctiveness among the item pools is known. In the present study we will introduce and illustrate the measure of item similarity for automatically generated items using a Natural Language Processing (NLP) technique so that the similarity among the set of generated items can be evaluated. To illustrate the measure, item models from two diverse content areas, Mathematics and Surgery, were used. Theoretical Framework Continuous testing is on important benefit of computer-based testing (CBT) which, in turn, requires enhanced test security to ensure that items are not overexposed. Traditional item development practices are resource intensive and are less viable for creating the content required with CBT. CBT requires large pools of operational test items. To address this daunting task of Establishing Item Uniqueness 3 developing large bank of test items, AIG research and practice has been developed over the last decade. For AIG, three general steps are required for developing items automatically (Gierl et al., 2012). First, the content experts create item model by identifying the features or elements in the assessment task that can be manipulated. Second, the item model is programmed for algorithmic variation of features or elements, using item generation software. Third, statistical models are used to estimate the psychometric properties of the generated items based on the combination of elements used in item assembly. The purpose of this study is to refine and extend step 2, the item generation process. Automatic generation is used to produce stem as well as options as part of the item text. The stem is a part of the item which contains the context, content, item, and/or the question the examinee is required to …